1 00:00:03,300 --> 00:00:09,350 [Music] 2 00:00:15,439 --> 00:00:13,009 I think many of us have had experiences 3 00:00:19,760 --> 00:00:15,449 at some time or another that might be 4 00:00:20,929 --> 00:00:19,770 characterized as beginner's luck Paulo I 5 00:00:22,670 --> 00:00:20,939 can never know I don't know how to 6 00:00:25,150 --> 00:00:22,680 pronounce his name quello the author of 7 00:00:27,889 --> 00:00:25,160 the bestseller The Alchemist I just 8 00:00:29,839 --> 00:00:27,899 suggested in his book that every search 9 00:00:31,519 --> 00:00:29,849 begins with beginner's luck and ends 10 00:00:33,650 --> 00:00:31,529 with the victors being severely tested 11 00:00:36,049 --> 00:00:33,660 and many of us I think have had 12 00:00:37,880 --> 00:00:36,059 experiences where when we try something 13 00:00:40,610 --> 00:00:37,890 new for the first time for whatever 14 00:00:44,330 --> 00:00:40,620 reason it seems to work remarkably well 15 00:00:47,030 --> 00:00:44,340 and then as we try to do it further it 16 00:00:48,260 --> 00:00:47,040 becomes more and more difficult I've 17 00:00:49,819 --> 00:00:48,270 personally experienced this and this 18 00:00:52,880 --> 00:00:49,829 will be part of my talk I've experienced 19 00:00:55,670 --> 00:00:52,890 this in in many realms of my own 20 00:00:57,590 --> 00:00:55,680 research but also I think even it seems 21 00:00:59,510 --> 00:00:57,600 as if there is a the at least the 22 00:01:01,520 --> 00:00:59,520 illusion that this happens in all realms 23 00:01:03,950 --> 00:01:01,530 of life there when we try we go to a 24 00:01:05,660 --> 00:01:03,960 place for the first time we try a new 25 00:01:08,120 --> 00:01:05,670 activity for the first time somehow it 26 00:01:09,499 --> 00:01:08,130 seems to really just open up to us and 27 00:01:12,590 --> 00:01:09,509 then as we keep working at it it's not 28 00:01:16,960 --> 00:01:12,600 as easy as it first appeared this is 29 00:01:20,450 --> 00:01:16,970 basic idea of beginner's luck seems to 30 00:01:23,330 --> 00:01:20,460 apply in research as well as certainly 31 00:01:26,899 --> 00:01:23,340 has been my experience it seems to be a 32 00:01:28,850 --> 00:01:26,909 ubiquitous aspect of research that our 33 00:01:31,670 --> 00:01:28,860 associate with a level of success that 34 00:01:35,660 --> 00:01:31,680 is then subsequently hard to replicate 35 00:01:37,789 --> 00:01:35,670 and I've observed this in a variety 36 00:01:40,760 --> 00:01:37,799 different ways but it was first noted by 37 00:01:42,679 --> 00:01:40,770 Rhine when he was attempting to 38 00:01:45,620 --> 00:01:42,689 characterize the drop in performance of 39 00:01:49,100 --> 00:01:45,630 one of his star clairvoyant participants 40 00:01:53,630 --> 00:01:49,110 who showed absolutely amazing ability to 41 00:01:56,749 --> 00:01:53,640 anticipate the nature of cards and then 42 00:02:00,170 --> 00:01:56,759 over time gradually diminished in his 43 00:02:02,920 --> 00:02:00,180 capacity to do this and this a basic 44 00:02:07,300 --> 00:02:02,930 notion of the decline effect has been 45 00:02:10,550 --> 00:02:07,310 recognized first and foremost in the SCI 46 00:02:13,160 --> 00:02:10,560 literature a Dean Radin in his book 47 00:02:15,649 --> 00:02:13,170 entangled Minds refers to it saying a 48 00:02:17,390 --> 00:02:15,659 frequent observation inside research is 49 00:02:19,280 --> 00:02:17,400 that when a new experiment is first 50 00:02:21,530 --> 00:02:19,290 conducted the outcomes are strikingly 51 00:02:23,540 --> 00:02:21,540 successful then as others try to 52 00:02:26,260 --> 00:02:23,550 replicate the effect they 53 00:02:29,450 --> 00:02:26,270 the effects they begin to fade and 54 00:02:31,040 --> 00:02:29,460 indeed and notably and this may be part 55 00:02:33,440 --> 00:02:31,050 of the secret of his success 56 00:02:35,810 --> 00:02:33,450 Dean Indyk has indicated he never tries 57 00:02:38,630 --> 00:02:35,820 to totally replicate prior findings as 58 00:02:41,510 --> 00:02:38,640 always you know changing the paradigm in 59 00:02:44,170 --> 00:02:41,520 particular ways and this may be a one of 60 00:02:46,670 --> 00:02:44,180 the critical secrets to a success and to 61 00:02:48,710 --> 00:02:46,680 others who have had a success in this 62 00:02:51,470 --> 00:02:48,720 field but it also may be one of the 63 00:02:56,690 --> 00:02:51,480 critical sources of why it's been so 64 00:02:59,000 --> 00:02:56,700 hard to get syphax to establish because 65 00:03:01,310 --> 00:02:59,010 when other people try to replicate the 66 00:03:04,160 --> 00:03:01,320 paradigms exactly they don't get the 67 00:03:07,130 --> 00:03:04,170 same degree of success as the original 68 00:03:10,850 --> 00:03:07,140 studies did um what I want to do today 69 00:03:13,160 --> 00:03:10,860 is to talk about examples of the decline 70 00:03:16,400 --> 00:03:13,170 effect both in my personal research and 71 00:03:19,780 --> 00:03:16,410 then also in other areas of Syre search 72 00:03:22,040 --> 00:03:19,790 in other areas of traditional 73 00:03:24,800 --> 00:03:22,050 conventional areas of research and then 74 00:03:28,790 --> 00:03:24,810 consider some of the different possible 75 00:03:30,770 --> 00:03:28,800 explanations that may be driving it so 76 00:03:32,300 --> 00:03:30,780 let me begin with my own personal 77 00:03:35,890 --> 00:03:32,310 experiences with the decline effect I 78 00:03:38,900 --> 00:03:35,900 started off as a graduate student 79 00:03:42,680 --> 00:03:38,910 investigating a phenomena that I came to 80 00:03:45,500 --> 00:03:42,690 call verbal overshadowing and verbal 81 00:03:47,210 --> 00:03:45,510 overshadowing is the finding that if you 82 00:03:50,210 --> 00:03:47,220 wear the paradigm worked as follows 83 00:03:51,949 --> 00:03:50,220 people would view say a videotape of a 84 00:03:53,150 --> 00:03:51,959 bank robbery and then one group of 85 00:03:55,100 --> 00:03:53,160 participants would be asked to describe 86 00:03:56,990 --> 00:03:55,110 in as much detail as possible the 87 00:03:58,370 --> 00:03:57,000 appearance of the face and another group 88 00:04:00,620 --> 00:03:58,380 of individuals would engage in an 89 00:04:03,410 --> 00:04:00,630 unrelated activity and then they were 90 00:04:05,210 --> 00:04:03,420 given a lineup now standard theories 91 00:04:06,680 --> 00:04:05,220 would suggest that engaging in verbal 92 00:04:09,020 --> 00:04:06,690 description would be a form of verbal 93 00:04:10,370 --> 00:04:09,030 rehearsal that this should be helpful 94 00:04:12,080 --> 00:04:10,380 and that the people who describe the 95 00:04:14,120 --> 00:04:12,090 face should be at least as good if not 96 00:04:17,020 --> 00:04:14,130 better than the people who didn't but we 97 00:04:19,159 --> 00:04:17,030 found was a rather substantial a 98 00:04:21,080 --> 00:04:19,169 negative effect such that when people 99 00:04:22,640 --> 00:04:21,090 describe the face they're actually less 100 00:04:24,710 --> 00:04:22,650 good at recognizing it than those 101 00:04:28,610 --> 00:04:24,720 individuals who engaged in an unrelated 102 00:04:32,360 --> 00:04:28,620 activity I replicated this effect five 103 00:04:33,500 --> 00:04:32,370 times in the original set of studies 104 00:04:36,980 --> 00:04:33,510 that were published in cognitive 105 00:04:39,310 --> 00:04:36,990 psychology and without difficulty was 106 00:04:41,630 --> 00:04:39,320 very easy to get the effect at that time 107 00:04:44,120 --> 00:04:41,640 subsequently as I continued to try to 108 00:04:45,920 --> 00:04:44,130 replicate this effect I found increasing 109 00:04:48,020 --> 00:04:45,930 difficulty I got it sometimes other 110 00:04:52,430 --> 00:04:48,030 times not the effects got much smaller I 111 00:04:54,860 --> 00:04:52,440 have a whole you know boxes of fail vert 112 00:04:57,590 --> 00:04:54,870 failed versions of this experiment and 113 00:05:00,590 --> 00:04:57,600 it was it was really very disheartening 114 00:05:02,120 --> 00:05:00,600 what I discovered was rather than trying 115 00:05:04,760 --> 00:05:02,130 to replicate verbal overshadowing effect 116 00:05:06,980 --> 00:05:04,770 with faces my best bet was to always try 117 00:05:08,300 --> 00:05:06,990 new areas and so we did it in a number 118 00:05:11,900 --> 00:05:08,310 of different domains we did with color 119 00:05:13,760 --> 00:05:11,910 we did it with tastes and shapes and 120 00:05:15,500 --> 00:05:13,770 various other things like that and what 121 00:05:18,710 --> 00:05:15,510 again what we found in these different 122 00:05:20,090 --> 00:05:18,720 areas is the first time we first a 123 00:05:22,280 --> 00:05:20,100 second time we get the effect but then 124 00:05:24,530 --> 00:05:22,290 as we tried to replicate that it would 125 00:05:25,820 --> 00:05:24,540 go away so we got it with music but that 126 00:05:27,620 --> 00:05:25,830 failed to replicate it we got it with 127 00:05:28,910 --> 00:05:27,630 color but that failed to replicate we 128 00:05:31,100 --> 00:05:28,920 got I don't have it listed here we got 129 00:05:32,780 --> 00:05:31,110 it with maps describing a map and then 130 00:05:34,580 --> 00:05:32,790 that failed to replicate so all these 131 00:05:36,500 --> 00:05:34,590 different cases were the first time we 132 00:05:38,090 --> 00:05:36,510 did an experiment or maybe the second we 133 00:05:40,160 --> 00:05:38,100 get the effect and then after that the 134 00:05:42,680 --> 00:05:40,170 effect we get smaller we also observe 135 00:05:44,420 --> 00:05:42,690 this not only with respect to nonverbal 136 00:05:46,940 --> 00:05:44,430 memories but also with respect to other 137 00:05:49,010 --> 00:05:46,950 sort of variations on the paradigm so in 138 00:05:51,980 --> 00:05:49,020 one series of studies these were four 139 00:05:54,650 --> 00:05:51,990 experiments published in JP general we 140 00:05:57,610 --> 00:05:54,660 had people think out loud while trying 141 00:05:59,750 --> 00:05:57,620 to solve insight problems or engage in 142 00:06:01,550 --> 00:05:59,760 silent solving so again the idea is that 143 00:06:03,650 --> 00:06:01,560 verbalization is somehow messing them up 144 00:06:05,810 --> 00:06:03,660 we got the effect quite robustly in 145 00:06:07,610 --> 00:06:05,820 those first series of four studies but 146 00:06:09,470 --> 00:06:07,620 then subsequently increasingly had 147 00:06:11,780 --> 00:06:09,480 difficulty replicating that effect in 148 00:06:13,940 --> 00:06:11,790 another study we used a paradigm known 149 00:06:16,070 --> 00:06:13,950 as implicit learning in which 150 00:06:17,660 --> 00:06:16,080 individuals are given artificial 151 00:06:19,400 --> 00:06:17,670 grammars and they have to learn new 152 00:06:21,710 --> 00:06:19,410 examples again we found when people 153 00:06:23,900 --> 00:06:21,720 fought out loud it disrupted performance 154 00:06:26,150 --> 00:06:23,910 subsequent replications failed to get 155 00:06:28,220 --> 00:06:26,160 that effect another paradigm was 156 00:06:30,530 --> 00:06:28,230 analogical retrieval where we had people 157 00:06:33,260 --> 00:06:30,540 think out loud while trying to remember 158 00:06:34,970 --> 00:06:33,270 what analogies were two stories that 159 00:06:37,550 --> 00:06:34,980 they'd heard before worked great at 160 00:06:41,660 --> 00:06:37,560 first and then dissipated so I've 161 00:06:43,430 --> 00:06:41,670 observed this fading effect just 162 00:06:44,900 --> 00:06:43,440 countless times in my mainstream 163 00:06:47,180 --> 00:06:44,910 research I'm just curious how many 164 00:06:49,190 --> 00:06:47,190 people in the audience who are 165 00:06:50,810 --> 00:06:49,200 researchers have also experienced a 166 00:06:53,030 --> 00:06:50,820 climb flex let's just get a raise 167 00:06:56,270 --> 00:06:53,040 of hands here okay so as you can see 168 00:06:57,830 --> 00:06:56,280 this is a how many people are just to 169 00:06:59,030 --> 00:06:57,840 get a sense how many people have done 170 00:07:02,530 --> 00:06:59,040 have gotten effects and they've never 171 00:07:04,910 --> 00:07:02,540 seen decline effects okay so you can see 172 00:07:06,500 --> 00:07:04,920 majority people who do research have 173 00:07:07,730 --> 00:07:06,510 experienced these decline effects it's 174 00:07:10,250 --> 00:07:07,740 important to you know get your base 175 00:07:12,440 --> 00:07:10,260 right here so this seems to be it's not 176 00:07:13,850 --> 00:07:12,450 just that you know sometimes some people 177 00:07:16,160 --> 00:07:13,860 publish the experiment and then other 178 00:07:18,740 --> 00:07:16,170 people don't the very same researcher 179 00:07:22,930 --> 00:07:18,750 can get the effect and then watch it 180 00:07:25,940 --> 00:07:22,940 dwindled so now I've also been doing a 181 00:07:27,830 --> 00:07:25,950 parapsychology research I'm it's it's 182 00:07:29,390 --> 00:07:27,840 somewhat unusual for somebody who's been 183 00:07:31,850 --> 00:07:29,400 doing the mainstream research to degree 184 00:07:34,730 --> 00:07:31,860 that I have to be doing this and I am 185 00:07:38,660 --> 00:07:34,740 totally aware of the fact that this does 186 00:07:41,330 --> 00:07:38,670 make me vulnerable to criticism among my 187 00:07:42,860 --> 00:07:41,340 mainstream colleagues but I still am 188 00:07:44,450 --> 00:07:42,870 persuaded that the evidence is 189 00:07:46,820 --> 00:07:44,460 sufficiently compelling that it's worth 190 00:07:48,590 --> 00:07:46,830 pursuing so I've been doing this I've 191 00:07:50,210 --> 00:07:48,600 looked at precognition in two different 192 00:07:51,830 --> 00:07:50,220 domains what I'd like to do is to tell 193 00:07:54,950 --> 00:07:51,840 you about both of those paradigms and 194 00:07:57,740 --> 00:07:54,960 then show you in both cases the decline 195 00:08:00,440 --> 00:07:57,750 effect the first paradigm both by the 196 00:08:03,410 --> 00:08:00,450 way both of these paradigms are closely 197 00:08:07,790 --> 00:08:03,420 related to a series of studies that just 198 00:08:09,410 --> 00:08:07,800 came out in Journal books JP SP one of 199 00:08:11,240 --> 00:08:09,420 the the flagship journal in social 200 00:08:13,040 --> 00:08:11,250 psychology by Daryl BEM many of you 201 00:08:16,910 --> 00:08:13,050 probably are familiar with that said I 202 00:08:19,730 --> 00:08:16,920 was a reviewer on that for that study 203 00:08:21,920 --> 00:08:19,740 and the basic inside of all those 204 00:08:24,320 --> 00:08:21,930 studies is that you basically have the 205 00:08:26,900 --> 00:08:24,330 independent variable happening after the 206 00:08:29,630 --> 00:08:26,910 dependent variable and so in his case 207 00:08:32,180 --> 00:08:29,640 there was a mere exposure happening 208 00:08:35,089 --> 00:08:32,190 after the people's judgments of the 209 00:08:38,000 --> 00:08:35,099 arousing quality of the effect and of 210 00:08:40,459 --> 00:08:38,010 the image but we did several variations 211 00:08:43,400 --> 00:08:40,469 on this one of them was temporarily 212 00:08:45,230 --> 00:08:43,410 reverse perceptual priming so there's 213 00:08:47,390 --> 00:08:45,240 been a lot of research which shows that 214 00:08:49,550 --> 00:08:47,400 if you view an image and then 215 00:08:52,000 --> 00:08:49,560 subsequently are flash the image very 216 00:08:55,220 --> 00:08:52,010 briefly that you can recognize the image 217 00:08:56,570 --> 00:08:55,230 more with a shorter interval if you'd 218 00:08:58,250 --> 00:08:56,580 seen it before there seems to be 219 00:09:00,350 --> 00:08:58,260 increased perceptual fluency to the 220 00:09:02,560 --> 00:09:00,360 image and as a result it takes less 221 00:09:04,610 --> 00:09:02,570 subsequent exposure to be able to 222 00:09:06,890 --> 00:09:04,620 identify it this is 223 00:09:08,300 --> 00:09:06,900 phenomena known as perceptual priming 224 00:09:10,460 --> 00:09:08,310 and so what we did is just to try to 225 00:09:12,350 --> 00:09:10,470 reverse the paradigm rather than showing 226 00:09:14,480 --> 00:09:12,360 people the image first and then having 227 00:09:16,430 --> 00:09:14,490 to make an identification from a briefly 228 00:09:17,270 --> 00:09:16,440 presented to see millisecond we did it 229 00:09:18,710 --> 00:09:17,280 the other way around 230 00:09:21,380 --> 00:09:18,720 we showed them the briefly presented 231 00:09:23,240 --> 00:09:21,390 stimulus first and asked them to make an 232 00:09:27,050 --> 00:09:23,250 identification and then subsequently 233 00:09:29,180 --> 00:09:27,060 either presented it again or not so the 234 00:09:32,300 --> 00:09:29,190 way the paradigm worked was as follows 235 00:09:33,650 --> 00:09:32,310 individuals viewed a fixation there was 236 00:09:35,150 --> 00:09:33,660 a noise mask I'll show you what that 237 00:09:37,670 --> 00:09:35,160 means in a moment then there was a 238 00:09:39,680 --> 00:09:37,680 briefly flashed image of the noise mask 239 00:09:42,170 --> 00:09:39,690 followed and they had indicate whether 240 00:09:43,610 --> 00:09:42,180 they knew what was presented and then 241 00:09:45,230 --> 00:09:43,620 the prime happened after they had 242 00:09:47,360 --> 00:09:45,240 already made the judgment either the 243 00:09:50,750 --> 00:09:47,370 image was repeated or was followed by a 244 00:09:53,510 --> 00:09:50,760 blank screen so there's the fixation 245 00:09:55,820 --> 00:09:53,520 point the noise mask was an image 246 00:09:57,680 --> 00:09:55,830 presented yes or no and then in this 247 00:09:59,180 --> 00:09:57,690 case an image is presented so that would 248 00:10:02,270 --> 00:09:59,190 be an example of a case in which there 249 00:10:05,900 --> 00:10:02,280 was a prime here's another case was an 250 00:10:08,690 --> 00:10:05,910 image presented yes or no and there's 251 00:10:11,630 --> 00:10:08,700 the blank screen so what we found is 252 00:10:15,380 --> 00:10:11,640 that when individuals have made the 253 00:10:17,750 --> 00:10:15,390 judgment and the image was followed by a 254 00:10:21,380 --> 00:10:17,760 prime they were significantly more 255 00:10:23,960 --> 00:10:21,390 likely to know say that they knew what 256 00:10:26,870 --> 00:10:23,970 the image was than if it was followed if 257 00:10:28,670 --> 00:10:26,880 it was not primed and we did this a 258 00:10:31,400 --> 00:10:28,680 number of different times we this is 259 00:10:34,240 --> 00:10:31,410 basically various different permutations 260 00:10:37,700 --> 00:10:34,250 on the basic paradigm we got the effect 261 00:10:40,310 --> 00:10:37,710 numerous times significantly but here's 262 00:10:43,010 --> 00:10:40,320 the decline effect which you can see is 263 00:10:45,830 --> 00:10:43,020 this is the magnitude of the prime 264 00:10:49,150 --> 00:10:45,840 effect and this is the experiment number 265 00:10:51,950 --> 00:10:49,160 and as you can see the effect size 266 00:10:53,960 --> 00:10:51,960 dwindled markedly with a subsequent 267 00:10:57,380 --> 00:10:53,970 experiments if you look at the overall 268 00:11:01,400 --> 00:10:57,390 linear effect that's here we get a 269 00:11:03,770 --> 00:11:01,410 p-value at the 0.0001 seven level of 270 00:11:06,650 --> 00:11:03,780 significance if you just cut it off here 271 00:11:08,180 --> 00:11:06,660 this is these are with smaller ends this 272 00:11:10,460 --> 00:11:08,190 is when we started running larger groups 273 00:11:12,940 --> 00:11:10,470 even when we just start from experiment 274 00:11:16,870 --> 00:11:12,950 seven to nineteen we still get a 275 00:11:18,889 --> 00:11:16,880 significant decline effect so we 276 00:11:20,989 --> 00:11:18,899 observed a very robust 277 00:11:24,049 --> 00:11:20,999 decline effect in this series of studies 278 00:11:26,239 --> 00:11:24,059 we ran over 2,000 participants that we 279 00:11:26,809 --> 00:11:26,249 really just hammered this experiment to 280 00:11:28,999 --> 00:11:26,819 the ground 281 00:11:34,489 --> 00:11:29,009 it really just washed it out completely 282 00:11:37,129 --> 00:11:34,499 eventually so the conclusions from this 283 00:11:39,559 --> 00:11:37,139 overall the effect remains significant 284 00:11:42,199 --> 00:11:39,569 but we observe this massive decline in 285 00:11:43,999 --> 00:11:42,209 significance the early studies may have 286 00:11:45,679 --> 00:11:44,009 had an advantage of smaller and and this 287 00:11:47,629 --> 00:11:45,689 is always a concern that you can have in 288 00:11:48,949 --> 00:11:47,639 these situations which it's when you 289 00:11:50,989 --> 00:11:48,959 have smaller and it's easier to get 290 00:11:53,269 --> 00:11:50,999 spurious results but even when we looked 291 00:11:55,129 --> 00:11:53,279 at the the later studies that had larger 292 00:11:58,309 --> 00:11:55,139 and we still get a significant decline 293 00:12:00,319 --> 00:11:58,319 effect so this is one compelling example 294 00:12:02,989 --> 00:12:00,329 I think of a decline effect in my own 295 00:12:05,090 --> 00:12:02,999 lab let me introduce you to a another 296 00:12:08,929 --> 00:12:05,100 paradigm this is a paradigm that was 297 00:12:11,629 --> 00:12:08,939 developed by my postdoc Michael Franklin 298 00:12:13,790 --> 00:12:11,639 who's been working in my lab for three 299 00:12:17,210 --> 00:12:13,800 years now I have to my hat goes off to 300 00:12:20,239 --> 00:12:17,220 him he got his PhD at University of 301 00:12:22,189 --> 00:12:20,249 Michigan and could have gotten a lots of 302 00:12:23,480 --> 00:12:22,199 really good mainstream postdocs that 303 00:12:26,329 --> 00:12:23,490 would not have threatened his career but 304 00:12:28,400 --> 00:12:26,339 rather was so committed to a pursuing 305 00:12:31,579 --> 00:12:28,410 SCI research that he came to join my lab 306 00:12:32,900 --> 00:12:31,589 and work on this paradigm I'm happy to 307 00:12:34,429 --> 00:12:32,910 also say that we've gotten i've also 308 00:12:36,189 --> 00:12:34,439 getting him doing mainstream research so 309 00:12:39,110 --> 00:12:36,199 i'm hoping that he's not necessarily 310 00:12:41,179 --> 00:12:39,120 destroyed his career by going in this 311 00:12:43,720 --> 00:12:41,189 direction but it is a real frustration 312 00:12:46,340 --> 00:12:43,730 of course that one has to really advise 313 00:12:48,139 --> 00:12:46,350 junior scientists against doing this 314 00:12:50,119 --> 00:12:48,149 kind of research because of the huge 315 00:12:52,009 --> 00:12:50,129 stigma that as you know is associated 316 00:12:53,780 --> 00:12:52,019 with this work so any of my hat goes off 317 00:12:56,049 --> 00:12:53,790 to him for having the guts to pursue 318 00:13:00,679 --> 00:12:56,059 this knowing the risks that he's taking 319 00:13:03,470 --> 00:13:00,689 so this paradigm is a retro causal 320 00:13:07,220 --> 00:13:03,480 practice paradigm so imagine that you 321 00:13:09,319 --> 00:13:07,230 gave individuals practice making a 322 00:13:13,489 --> 00:13:09,329 particular kind of perceptual 323 00:13:15,199 --> 00:13:13,499 identification and then you gave them 324 00:13:16,549 --> 00:13:15,209 two different perceptual identification 325 00:13:18,109 --> 00:13:16,559 tasks one with the items they had 326 00:13:19,069 --> 00:13:18,119 practiced with and know that they'd 327 00:13:21,739 --> 00:13:19,079 never seen before 328 00:13:23,869 --> 00:13:21,749 you'd be able to tell based on the 329 00:13:25,519 --> 00:13:23,879 performance at time to which ones they 330 00:13:26,929 --> 00:13:25,529 had practice with the time one because 331 00:13:30,769 --> 00:13:26,939 the performance should be different and 332 00:13:32,420 --> 00:13:30,779 so that's basically the the idea here 333 00:13:34,519 --> 00:13:32,430 except it's reversed 334 00:13:36,530 --> 00:13:34,529 in this case individuals make 335 00:13:39,079 --> 00:13:36,540 discriminations with two sets of stimuli 336 00:13:41,450 --> 00:13:39,089 initially and then subsequently they get 337 00:13:44,329 --> 00:13:41,460 additional practice with just one or the 338 00:13:49,880 --> 00:13:44,339 other set of stimuli the stimuli that he 339 00:13:52,880 --> 00:13:49,890 used are these nonverbal shapes and so 340 00:13:55,610 --> 00:13:52,890 people are basically asked to respond to 341 00:13:57,920 --> 00:13:55,620 either one kind of shape or another type 342 00:14:00,530 --> 00:13:57,930 of shape so it's a basic perceptual 343 00:14:02,720 --> 00:14:00,540 discrimination task and the pattern of 344 00:14:04,579 --> 00:14:02,730 results is a little bit quirky it's not 345 00:14:07,150 --> 00:14:04,589 just that practice with one with the 346 00:14:11,420 --> 00:14:07,160 shape that you've seen before leads to 347 00:14:14,870 --> 00:14:11,430 improve performance but rather what he 348 00:14:21,370 --> 00:14:14,880 found was a significant priming effect 349 00:14:26,930 --> 00:14:21,380 such that practice with shape a led to 350 00:14:29,210 --> 00:14:26,940 superior performance with shape a but it 351 00:14:32,000 --> 00:14:29,220 also practiced with a shape a also led 352 00:14:34,310 --> 00:14:32,010 to a superior performance with shape B 353 00:14:36,590 --> 00:14:34,320 so for some reason it's of the pattern 354 00:14:39,079 --> 00:14:36,600 of results is a little peculiar but the 355 00:14:42,110 --> 00:14:39,089 basic point is that he's finding is that 356 00:14:44,060 --> 00:14:42,120 practice at time two influenced 357 00:14:46,280 --> 00:14:44,070 performance at time one that's the 358 00:14:49,100 --> 00:14:46,290 take-home message that this paradigm 359 00:14:51,650 --> 00:14:49,110 revealed and he found that pattern in 360 00:14:54,620 --> 00:14:51,660 both the original study and a number of 361 00:14:58,130 --> 00:14:54,630 different replications now he presented 362 00:15:00,110 --> 00:14:58,140 this work at the the major Brown Bag at 363 00:15:01,730 --> 00:15:00,120 our department and people were sure 364 00:15:03,920 --> 00:15:01,740 there must be something about the random 365 00:15:05,480 --> 00:15:03,930 assignment that that maybe because the 366 00:15:07,040 --> 00:15:05,490 computer was doing the assignment even 367 00:15:09,590 --> 00:15:07,050 if it was using a random number 368 00:15:11,120 --> 00:15:09,600 generator that was based on physics 369 00:15:13,750 --> 00:15:11,130 property there must be a problem there 370 00:15:16,449 --> 00:15:13,760 and so what he did instead of using a 371 00:15:21,079 --> 00:15:16,459 random number generator based on the 372 00:15:26,180 --> 00:15:21,089 computer was instead to use a outcome of 373 00:15:27,470 --> 00:15:26,190 a off-site roulette wheel so if you 374 00:15:31,120 --> 00:15:27,480 think about this this is very promising 375 00:15:37,220 --> 00:15:31,130 so in phase one individuals are given 376 00:15:41,960 --> 00:15:37,230 practice with they work with both J pay 377 00:15:44,810 --> 00:15:41,970 and shape B okay and then at time two 378 00:15:46,310 --> 00:15:44,820 there is a roulette wheel spin which 379 00:15:48,440 --> 00:15:46,320 determines which shape the 380 00:15:51,110 --> 00:15:48,450 which set of shapes are going to get 381 00:15:53,330 --> 00:15:51,120 additional practice with and so if the 382 00:15:55,640 --> 00:15:53,340 ball lands on a red they get practice 383 00:15:58,010 --> 00:15:55,650 with shape a and if the ball lands on 384 00:16:00,920 --> 00:15:58,020 black they get practice with shape E now 385 00:16:02,570 --> 00:16:00,930 the nice thing about this is what this 386 00:16:04,700 --> 00:16:02,580 means is that you should be able to 387 00:16:06,950 --> 00:16:04,710 predict from their performance on phase 388 00:16:08,900 --> 00:16:06,960 one what the roulette wheel is going to 389 00:16:11,360 --> 00:16:08,910 spin to if it's going to if their 390 00:16:13,010 --> 00:16:11,370 performance suggests that there that 391 00:16:15,320 --> 00:16:13,020 there is more like that they are going 392 00:16:16,820 --> 00:16:15,330 to have practice with shape a then that 393 00:16:19,490 --> 00:16:16,830 means the roulette wheel is going to 394 00:16:20,780 --> 00:16:19,500 assign them based on red and if it looks 395 00:16:22,240 --> 00:16:20,790 more like B it suggests that the 396 00:16:24,800 --> 00:16:22,250 roulette wheel is going to be associated 397 00:16:27,130 --> 00:16:24,810 with black so the exciting thing about 398 00:16:31,160 --> 00:16:27,140 this paradigm basically is we are now 399 00:16:34,430 --> 00:16:31,170 predicting the outcome of a offsite 400 00:16:37,430 --> 00:16:34,440 roulette wheel now money speaks right so 401 00:16:39,890 --> 00:16:37,440 if this paradigm were to hold up we 402 00:16:42,080 --> 00:16:39,900 would need to get grant money right we 403 00:16:44,300 --> 00:16:42,090 could just make our money for the 404 00:16:46,220 --> 00:16:44,310 research of winning money off of a 405 00:16:50,660 --> 00:16:46,230 roulette wheel so this is a very 406 00:16:52,760 --> 00:16:50,670 exciting a procedure and obviously you 407 00:16:54,470 --> 00:16:52,770 know skeptics are gonna have a hard time 408 00:16:55,670 --> 00:16:54,480 going well yes I know you made millions 409 00:16:57,890 --> 00:16:55,680 of dollars but you know who was 410 00:17:00,710 --> 00:16:57,900 confounded right so clearly this is a 411 00:17:02,180 --> 00:17:00,720 very very compelling kind of paradigm 412 00:17:03,500 --> 00:17:02,190 for persuading people that there's 413 00:17:07,970 --> 00:17:03,510 really something there because money 414 00:17:10,010 --> 00:17:07,980 speaks um so the basic finding and this 415 00:17:13,100 --> 00:17:10,020 is what's basically happening is our 416 00:17:14,990 --> 00:17:13,110 ability to predict based on performance 417 00:17:17,300 --> 00:17:15,000 at time one whether or not the roulette 418 00:17:20,660 --> 00:17:17,310 wheel is going to assign people to one 419 00:17:22,310 --> 00:17:20,670 condition or the or the other and the 420 00:17:25,130 --> 00:17:22,320 refining x' with a hundred and fifty 421 00:17:26,660 --> 00:17:25,140 three participants the performance our 422 00:17:32,000 --> 00:17:26,670 ability to predict the spin of the 423 00:17:33,620 --> 00:17:32,010 roulette wheel was as 55 0.55 424 00:17:35,180 --> 00:17:33,630 we're chances point four eight six and 425 00:17:37,370 --> 00:17:35,190 the recent chance is not 50s because 426 00:17:40,340 --> 00:17:37,380 this is a gambling site and so there's a 427 00:17:42,580 --> 00:17:40,350 one green where you don't win so he's 428 00:17:45,320 --> 00:17:42,590 six point nine above chance which is 429 00:17:46,910 --> 00:17:45,330 significant point one Oh with the two 430 00:17:51,560 --> 00:17:46,920 tailed test 431 00:17:55,820 --> 00:17:51,570 so marginally significant importantly as 432 00:18:00,019 --> 00:17:55,830 you can see here the or them it's better 433 00:18:02,839 --> 00:18:00,029 to see it here this is the magnitude 434 00:18:04,519 --> 00:18:02,849 the effect over groups of ten subjects 435 00:18:06,739 --> 00:18:04,529 and basically what we got 436 00:18:09,669 --> 00:18:06,749 was very significant in the first 437 00:18:11,989 --> 00:18:09,679 quartile and then less significant 438 00:18:15,589 --> 00:18:11,999 subsequently and if you look at the 439 00:18:20,359 --> 00:18:15,599 overall decline effect its are negative 440 00:18:23,320 --> 00:18:20,369 0.45 P equals o 5 so at the moment this 441 00:18:25,039 --> 00:18:23,330 effect seems to also have decline now 442 00:18:27,169 --> 00:18:25,049 Michael and I have somewhat different 443 00:18:30,229 --> 00:18:27,179 perspectives on this he's hoping that 444 00:18:32,329 --> 00:18:30,239 the effect this is just a lag and that 445 00:18:35,060 --> 00:18:32,339 it'll come back and I'm seeing this more 446 00:18:36,799 --> 00:18:35,070 as there's the decline effect again but 447 00:18:38,469 --> 00:18:36,809 there seems to be a number of things 448 00:18:40,909 --> 00:18:38,479 that we can conclude from this 449 00:18:42,829 --> 00:18:40,919 experiment the first is that overall 450 00:18:44,269 --> 00:18:42,839 effective Retro causal effect practice 451 00:18:45,709 --> 00:18:44,279 remains significant it's important to 452 00:18:49,639 --> 00:18:45,719 keep in mind that this is just one of a 453 00:18:51,289 --> 00:18:49,649 whole series of over 900 participants 454 00:18:53,329 --> 00:18:51,299 that Michael's run in this paradigm and 455 00:18:56,779 --> 00:18:53,339 overall we still have a significant 456 00:18:59,359 --> 00:18:56,789 retro causal practice effect secondly 457 00:19:00,529 --> 00:18:59,369 this is a really elegant paradigm and I 458 00:19:02,389 --> 00:19:00,539 would encourage people to think about 459 00:19:07,430 --> 00:19:02,399 developing their paradigms in this way 460 00:19:09,169 --> 00:19:07,440 in that it the obvious significance of 461 00:19:11,959 --> 00:19:09,179 such a finding if you can find a finding 462 00:19:13,639 --> 00:19:11,969 that can you know make money then that's 463 00:19:14,329 --> 00:19:13,649 really going to persuade skeptics in a 464 00:19:16,879 --> 00:19:14,339 way then that 465 00:19:18,950 --> 00:19:16,889 little else could and then finally 466 00:19:22,219 --> 00:19:18,960 relevant to the current discussion here 467 00:19:24,769 --> 00:19:22,229 the decline effect observed was 468 00:19:29,329 --> 00:19:24,779 significant so again we are seeing this 469 00:19:31,369 --> 00:19:29,339 haunting decline effect all right so um 470 00:19:34,579 --> 00:19:31,379 it turns out that decline effects are 471 00:19:36,349 --> 00:19:34,589 not just limited no sorry before I get 472 00:19:38,979 --> 00:19:36,359 into that the client in fact is been 473 00:19:42,169 --> 00:19:38,989 shown up in a number of meta analyses of 474 00:19:44,119 --> 00:19:42,179 in various areas of SCI research so this 475 00:19:46,269 --> 00:19:44,129 is a meta-analysis reported by dick 476 00:19:49,489 --> 00:19:46,279 Biermann in 2001 on dice throwing 477 00:19:53,029 --> 00:19:49,499 showing a significant decline in 478 00:19:55,310 --> 00:19:53,039 people's ability to get dice to produce 479 00:19:59,299 --> 00:19:55,320 the outcomes that they were trying to 480 00:20:02,299 --> 00:19:59,309 get here is a similar decline effect in 481 00:20:03,529 --> 00:20:02,309 the ganzfeld procedure presumably 482 00:20:06,829 --> 00:20:03,539 everyone here knows the gone sale 483 00:20:09,469 --> 00:20:06,839 procedure is basically a remote it's 484 00:20:11,810 --> 00:20:09,479 it's a telepathy procedure where 485 00:20:13,730 --> 00:20:11,820 individuals are able to imagine know 486 00:20:16,910 --> 00:20:13,740 what the sender is 487 00:20:18,500 --> 00:20:16,920 sending with half ping-pong balls over 488 00:20:22,240 --> 00:20:18,510 their eyes to get sort of a better sense 489 00:20:24,770 --> 00:20:22,250 of the image that's being sent again a 490 00:20:27,290 --> 00:20:24,780 significant decline effect but 491 00:20:29,630 --> 00:20:27,300 interestingly there seems to be what may 492 00:20:31,340 --> 00:20:29,640 be a decline so here they've 493 00:20:33,020 --> 00:20:31,350 demonstrated a Cline effect and this 494 00:20:34,880 --> 00:20:33,030 race is this really interesting thing if 495 00:20:37,700 --> 00:20:34,890 you demonstrate empirically the decline 496 00:20:39,410 --> 00:20:37,710 effect does that replicate well it turns 497 00:20:44,140 --> 00:20:39,420 out that it seems that even the decline 498 00:20:50,110 --> 00:20:44,150 effect is subject to the decline effect 499 00:20:53,090 --> 00:20:50,120 so here is a recent here is a recent 500 00:20:54,590 --> 00:20:53,100 meta-analysis by a store minal and you 501 00:20:56,030 --> 00:20:54,600 can see right here the sort of the peak 502 00:20:57,919 --> 00:20:56,040 of the client effect is right you know 503 00:20:59,570 --> 00:20:57,929 in this Biermann meta-analysis and now 504 00:21:02,000 --> 00:20:59,580 it comes back so the evening things out 505 00:21:03,770 --> 00:21:02,010 so it may be my sort of secret hope here 506 00:21:05,840 --> 00:21:03,780 is that by really going after the 507 00:21:09,590 --> 00:21:05,850 decline effect I can get rid of it once 508 00:21:11,620 --> 00:21:09,600 and for all and by the way this is not 509 00:21:14,419 --> 00:21:11,630 the only place where this you get this 510 00:21:17,630 --> 00:21:14,429 decline in return it's also been found 511 00:21:19,660 --> 00:21:17,640 in the RNG paradigm again you see this 512 00:21:21,799 --> 00:21:19,670 decline effect and then a return 513 00:21:23,960 --> 00:21:21,809 oftentimes the return is not quite as 514 00:21:26,090 --> 00:21:23,970 strong as it never seems to quite get 515 00:21:28,220 --> 00:21:26,100 back to its original place but there 516 00:21:31,280 --> 00:21:28,230 does seem to be this peculiar decline in 517 00:21:33,140 --> 00:21:31,290 the decline effect all right 518 00:21:34,430 --> 00:21:33,150 so conclusions the meta-analysis of psy 519 00:21:37,010 --> 00:21:34,440 research the client effect is observed 520 00:21:38,690 --> 00:21:37,020 in a number of domains after decline a 521 00:21:41,240 --> 00:21:38,700 return is observed in the longest 522 00:21:43,430 --> 00:21:41,250 studied domains and could this plausibly 523 00:21:45,950 --> 00:21:43,440 could be a decline in the decline effect 524 00:21:48,200 --> 00:21:45,960 now turns out that this is not as 525 00:21:49,940 --> 00:21:48,210 already anticipated by my discussions of 526 00:21:52,250 --> 00:21:49,950 my own research the only place where 527 00:21:55,760 --> 00:21:52,260 decline effects have been observed they 528 00:21:57,490 --> 00:21:55,770 have also been observed in a number of 529 00:21:59,600 --> 00:21:57,500 different domains this is a 530 00:22:00,380 --> 00:21:59,610 meta-analysis of drug treatments of 531 00:22:03,380 --> 00:22:00,390 schizophrenia 532 00:22:05,450 --> 00:22:03,390 actually my mother oddly enough was an 533 00:22:07,460 --> 00:22:05,460 author on this meta-analysis and I I 534 00:22:09,169 --> 00:22:07,470 just was at the house one day and I saw 535 00:22:10,940 --> 00:22:09,179 this you know why did drug effects 536 00:22:13,070 --> 00:22:10,950 decline I was like oh my god it's there 537 00:22:15,620 --> 00:22:13,080 too and so they have this really 538 00:22:18,020 --> 00:22:15,630 powerful the client affected 0.55 in 539 00:22:20,240 --> 00:22:18,030 this many of you may know that there was 540 00:22:21,890 --> 00:22:20,250 an article that was described in by 541 00:22:23,690 --> 00:22:21,900 Jonah Lehrer or article written by Jonah 542 00:22:26,600 --> 00:22:23,700 Lehrer in The New Yorker that got a lot 543 00:22:27,680 --> 00:22:26,610 of press I got a lot of emails as a 544 00:22:30,470 --> 00:22:27,690 result of that 545 00:22:32,930 --> 00:22:30,480 one of the emails I got was from Robert 546 00:22:35,330 --> 00:22:32,940 Kaplan who is notably 547 00:22:37,010 --> 00:22:35,340 now the aimh Associate Director for 548 00:22:39,650 --> 00:22:37,020 behavioral and social sciences so a 549 00:22:42,800 --> 00:22:39,660 major player in mainstream science and 550 00:22:46,010 --> 00:22:42,810 he sent me several meta analyses 551 00:22:48,140 --> 00:22:46,020 demonstrating decline effects in a 552 00:22:50,150 --> 00:22:48,150 variety of different drug research 553 00:22:54,860 --> 00:22:50,160 domains including previs 10 which is a 554 00:23:01,100 --> 00:22:54,870 treatment for cholesterol timolol which 555 00:23:02,510 --> 00:23:01,110 is a beta blocker and also got Latin 556 00:23:05,510 --> 00:23:02,520 approach which is a treatment of 557 00:23:07,070 --> 00:23:05,520 glaucoma I'm delighted to say that dr. 558 00:23:09,770 --> 00:23:07,080 Kaplan is working on getting an 559 00:23:11,750 --> 00:23:09,780 invitation for me to speak at the NIH so 560 00:23:13,460 --> 00:23:11,760 there is this hope that the decline 561 00:23:15,290 --> 00:23:13,470 effect may be sort of an entry point for 562 00:23:18,200 --> 00:23:15,300 getting a finding that was originally 563 00:23:20,030 --> 00:23:18,210 discovered in parapsychology really into 564 00:23:22,310 --> 00:23:20,040 the mainstream this decline effect has 565 00:23:25,190 --> 00:23:22,320 also been reported in men analyses in 566 00:23:27,890 --> 00:23:25,200 biology there's a meta-analysis by Jenny 567 00:23:29,150 --> 00:23:27,900 ins and Moeller and they say in which 568 00:23:30,890 --> 00:23:29,160 they found there was a small but 569 00:23:32,900 --> 00:23:30,900 significant decline effect in size with 570 00:23:34,280 --> 00:23:32,910 your publication and the effect of your 571 00:23:36,470 --> 00:23:34,290 publication remained even after we 572 00:23:38,720 --> 00:23:36,480 controlled for sampling effort similar 573 00:23:40,760 --> 00:23:38,730 to my study where we had smaller and at 574 00:23:43,700 --> 00:23:40,770 first and larger later on same thing as 575 00:23:45,940 --> 00:23:43,710 observed in this case and these are just 576 00:23:48,500 --> 00:23:45,950 the numbers that bear that out 577 00:23:50,270 --> 00:23:48,510 so are the conclusions from the decline 578 00:23:51,590 --> 00:23:50,280 effect in mainstream science there's 579 00:23:52,730 --> 00:23:51,600 been declining effects observed in a 580 00:23:54,290 --> 00:23:52,740 variety of means including any 581 00:23:56,090 --> 00:23:54,300 psychotics cholesterol-lowering drugs 582 00:23:57,830 --> 00:23:56,100 beta blockers treatment of glaucoma and 583 00:24:01,240 --> 00:23:57,840 in a variety of different domains of 584 00:24:03,830 --> 00:24:01,250 biology this is not just limited to a 585 00:24:05,930 --> 00:24:03,840 research although clearly there's some 586 00:24:07,640 --> 00:24:05,940 constraints here I mean balls when you 587 00:24:10,040 --> 00:24:07,650 drop them they fall in exactly the same 588 00:24:12,790 --> 00:24:10,050 rate every time so there's a lot that we 589 00:24:14,990 --> 00:24:12,800 don't understand about what the specific 590 00:24:16,660 --> 00:24:15,000 parameters are for where decline effects 591 00:24:20,330 --> 00:24:16,670 are observed and where they're not 592 00:24:21,830 --> 00:24:20,340 alright now let me take a breath step 593 00:24:23,570 --> 00:24:21,840 back for a moment and talk about 594 00:24:26,540 --> 00:24:23,580 possible accounts of the client effect 595 00:24:28,390 --> 00:24:26,550 it's really critical to emphasize that 596 00:24:30,560 --> 00:24:28,400 there are a number of very compelling 597 00:24:32,600 --> 00:24:30,570 conventional accounts of decline effects 598 00:24:34,400 --> 00:24:32,610 and you know before we get all carried 599 00:24:36,680 --> 00:24:34,410 away with the side kinds of accounts 600 00:24:38,630 --> 00:24:36,690 it's really very careful to acknowledge 601 00:24:40,490 --> 00:24:38,640 conventional accounts think about them 602 00:24:41,510 --> 00:24:40,500 carefully and consider to what degree 603 00:24:44,090 --> 00:24:41,520 they may be playing 604 00:24:47,240 --> 00:24:44,100 the first one most obviously is 605 00:24:48,740 --> 00:24:47,250 regression to the mean that is every now 606 00:24:50,570 --> 00:24:48,750 and then when you run or when you run 607 00:24:52,640 --> 00:24:50,580 study sometimes you're going to get the 608 00:24:53,840 --> 00:24:52,650 actual variance associated with the 609 00:24:55,370 --> 00:24:53,850 manipulation and there's also going to 610 00:24:57,140 --> 00:24:55,380 be noise variance and when the noise 611 00:24:59,540 --> 00:24:57,150 variance goes in the same direction as 612 00:25:01,160 --> 00:24:59,550 the actual variance that you're looking 613 00:25:05,150 --> 00:25:01,170 for that's going to lead to a distortion 614 00:25:07,580 --> 00:25:05,160 a distorted estimate of the effect size 615 00:25:10,160 --> 00:25:07,590 and so it may be that if you get lucky 616 00:25:12,260 --> 00:25:10,170 and get this distortion that that's 617 00:25:14,510 --> 00:25:12,270 going to lead you to overestimate the 618 00:25:16,549 --> 00:25:14,520 effect size initially it's a very 619 00:25:19,030 --> 00:25:16,559 compelling explanation one thing it 620 00:25:21,410 --> 00:25:19,040 doesn't really explain is why the 621 00:25:22,850 --> 00:25:21,420 decline effect tends to have this linear 622 00:25:24,530 --> 00:25:22,860 pattern regression to the mean would 623 00:25:26,419 --> 00:25:24,540 predict that you should get a big effect 624 00:25:28,490 --> 00:25:26,429 and then it should just vary around the 625 00:25:29,240 --> 00:25:28,500 mean thereafter so that's the problem 626 00:25:31,430 --> 00:25:29,250 with that one 627 00:25:33,560 --> 00:25:31,440 definitely can account for some findings 628 00:25:35,240 --> 00:25:33,570 but not necessarily all of them another 629 00:25:37,700 --> 00:25:35,250 one I think this is a very very elegant 630 00:25:39,410 --> 00:25:37,710 explanation it was suggested to me by my 631 00:25:41,630 --> 00:25:39,420 colleague dance Millikin University of 632 00:25:43,130 --> 00:25:41,640 Waterloo it may be that when we first 633 00:25:45,169 --> 00:25:43,140 when we run studies there are a number 634 00:25:47,150 --> 00:25:45,179 of aspects to the paradigm that we have 635 00:25:48,410 --> 00:25:47,160 made explicit that we know are important 636 00:25:50,210 --> 00:25:48,420 but there are other aspects of the 637 00:25:52,130 --> 00:25:50,220 paradigm that may be important but we 638 00:25:54,230 --> 00:25:52,140 haven't recognized them as such so when 639 00:25:55,549 --> 00:25:54,240 we replicate the experiment we always 640 00:25:56,960 --> 00:25:55,559 make sure to do the parts that we know 641 00:25:58,549 --> 00:25:56,970 are important but they're all these 642 00:26:00,350 --> 00:25:58,559 other parts that what we thought are 643 00:26:02,210 --> 00:26:00,360 superfluous that actually turn out to be 644 00:26:04,430 --> 00:26:02,220 critical because we don't recognizing 645 00:26:06,799 --> 00:26:04,440 recognized them as being critical as we 646 00:26:08,510 --> 00:26:06,809 replicate they gradually become less 647 00:26:10,430 --> 00:26:08,520 included and so the effect size 648 00:26:12,080 --> 00:26:10,440 dissipates because we're not recognizing 649 00:26:13,820 --> 00:26:12,090 some of the critical variables this 650 00:26:16,190 --> 00:26:13,830 makes a lot of sense but erases the 651 00:26:18,020 --> 00:26:16,200 question why do we get so lucky and get 652 00:26:18,770 --> 00:26:18,030 it all right at first and then go in the 653 00:26:20,540 --> 00:26:18,780 wrong direction 654 00:26:22,549 --> 00:26:20,550 shouldn't it also happen that you know 655 00:26:25,010 --> 00:26:22,559 we get it wrong at first and then get 656 00:26:27,200 --> 00:26:25,020 better as time goes on so it makes sense 657 00:26:28,720 --> 00:26:27,210 but it's this has a sort of peculiar 658 00:26:31,370 --> 00:26:28,730 reliance on luck 659 00:26:32,600 --> 00:26:31,380 another possibility is just that we 660 00:26:35,570 --> 00:26:32,610 refine the procedure that there 661 00:26:38,360 --> 00:26:35,580 confounds in the procedures initially 662 00:26:40,040 --> 00:26:38,370 but that then as we refine those 663 00:26:41,540 --> 00:26:40,050 procedures we get rid of the confounds 664 00:26:43,040 --> 00:26:41,550 and that makes the effect go away well 665 00:26:45,080 --> 00:26:43,050 that makes sense but shouldn't we also 666 00:26:46,820 --> 00:26:45,090 be refining procedures and get stronger 667 00:26:47,990 --> 00:26:46,830 yeah we should understand the phenomena 668 00:26:50,060 --> 00:26:48,000 better and actually be able to get the 669 00:26:51,980 --> 00:26:50,070 effects better so this makes sense but 670 00:26:55,280 --> 00:26:51,990 it's a little bit confusing as to why it 671 00:26:57,680 --> 00:26:55,290 would only go in one direction and 672 00:26:59,630 --> 00:26:57,690 lastly the possibility publication bias 673 00:27:00,980 --> 00:26:59,640 that um initially the only way you can 674 00:27:02,780 --> 00:27:00,990 get something published is if you get a 675 00:27:04,640 --> 00:27:02,790 large effect and if it's very sort of 676 00:27:06,260 --> 00:27:04,650 straightforward but then as people are 677 00:27:08,240 --> 00:27:06,270 attempting to replicate it or looking at 678 00:27:10,010 --> 00:27:08,250 various different parameters there may 679 00:27:12,170 --> 00:27:10,020 not be the same onus to have to get such 680 00:27:13,430 --> 00:27:12,180 a large effect size this also makes a 681 00:27:14,960 --> 00:27:13,440 lot of sense this this has been the 682 00:27:17,210 --> 00:27:14,970 explanation that Virginians 683 00:27:19,430 --> 00:27:17,220 and colleagues suggested but it doesn't 684 00:27:20,840 --> 00:27:19,440 explain my own experience I didn't 685 00:27:22,760 --> 00:27:20,850 there's no publication bias to produce 686 00:27:25,610 --> 00:27:22,770 that you know linear decline in the 687 00:27:27,410 --> 00:27:25,620 however many fifteen precognition cities 688 00:27:29,990 --> 00:27:27,420 so so that alone doesn't explain at all 689 00:27:31,640 --> 00:27:30,000 now it's possible and I would have to 690 00:27:34,370 --> 00:27:31,650 say you know when I put it on my 691 00:27:37,280 --> 00:27:34,380 traditional I don't have to save it the 692 00:27:39,080 --> 00:27:37,290 most parsimonious explanation is it some 693 00:27:40,760 --> 00:27:39,090 combination of all of these that that 694 00:27:41,930 --> 00:27:40,770 any one of these alone doesn't do it but 695 00:27:44,630 --> 00:27:41,940 when you combine them all in just the 696 00:27:46,520 --> 00:27:44,640 right way that's sufficient to account 697 00:27:48,110 --> 00:27:46,530 for the client effect and I think that 698 00:27:51,020 --> 00:27:48,120 is the explanation that needs to be beat 699 00:27:53,030 --> 00:27:51,030 that said I also think and I've talked 700 00:27:54,980 --> 00:27:53,040 about this I've stuck my neck out and 701 00:27:57,290 --> 00:27:54,990 talked about this you know publicly and 702 00:27:59,600 --> 00:27:57,300 even in an article in nature and in my 703 00:28:02,840 --> 00:27:59,610 interview with Jonah Lehrer I think it's 704 00:28:04,970 --> 00:28:02,850 also appropriate to consider if not to 705 00:28:06,860 --> 00:28:04,980 subscribe yet to subscribe to some 706 00:28:09,410 --> 00:28:06,870 non-conventional accounts of the decline 707 00:28:11,120 --> 00:28:09,420 effect and so here I'm going to sort of 708 00:28:13,190 --> 00:28:11,130 let my hair down a little bit and talk 709 00:28:14,780 --> 00:28:13,200 about what might be some things going on 710 00:28:18,830 --> 00:28:14,790 here again I want to emphasize that I 711 00:28:21,020 --> 00:28:18,840 see these is incredibly speculative one 712 00:28:23,870 --> 00:28:21,030 possibility is that there's some sort of 713 00:28:26,960 --> 00:28:23,880 Heisenberg in process going on here that 714 00:28:30,410 --> 00:28:26,970 somehow be in the process of observation 715 00:28:32,210 --> 00:28:30,420 of scientific phenomena we changed the 716 00:28:34,220 --> 00:28:32,220 phenomena so it's typically the way 717 00:28:36,490 --> 00:28:34,230 Heisenberg and the Copenhagen account 718 00:28:39,620 --> 00:28:36,500 goes and mind you this is no longer the 719 00:28:42,350 --> 00:28:39,630 mostly accepted version the act of 720 00:28:46,850 --> 00:28:42,360 observation changes at the local level 721 00:28:48,770 --> 00:28:46,860 the particular way in which it collapses 722 00:28:49,970 --> 00:28:48,780 the probability cloud and influences the 723 00:28:53,540 --> 00:28:49,980 particular manner in which say an 724 00:28:55,520 --> 00:28:53,550 electron is located here the idea is 725 00:28:57,320 --> 00:28:55,530 that it's not influencing the specific 726 00:28:58,430 --> 00:28:57,330 location of a particular event or 727 00:29:01,220 --> 00:28:58,440 occurrence of a particular event but 728 00:29:04,580 --> 00:29:01,230 rather the nature of the rules of 729 00:29:06,830 --> 00:29:04,590 science more generally so this sort of 730 00:29:08,840 --> 00:29:06,840 assumes it challenges what is an 731 00:29:10,670 --> 00:29:08,850 absolutely fundamental premise 732 00:29:11,900 --> 00:29:10,680 of a current science which is that the 733 00:29:14,090 --> 00:29:11,910 laws of nature whatever they are 734 00:29:16,550 --> 00:29:14,100 discovered or undiscovered or immutable 735 00:29:18,680 --> 00:29:16,560 and this basically says no the process 736 00:29:22,460 --> 00:29:18,690 of observation may actually change the 737 00:29:24,050 --> 00:29:22,470 principles of science now this is very 738 00:29:25,640 --> 00:29:24,060 radical but it's also important to 739 00:29:28,630 --> 00:29:25,650 emphasize that I'm not the only one who 740 00:29:31,040 --> 00:29:28,640 suggested this physicists have also 741 00:29:33,500 --> 00:29:31,050 acknowledged that the premise that the 742 00:29:36,050 --> 00:29:33,510 laws of reality are immutable is an 743 00:29:38,980 --> 00:29:36,060 assumption and cannot be taken as you 744 00:29:41,930 --> 00:29:38,990 necessarily dogma 745 00:29:43,820 --> 00:29:41,940 so some possible things that might 746 00:29:46,490 --> 00:29:43,830 contribute to this way in which the 747 00:29:48,140 --> 00:29:46,500 active observation has its effect one 748 00:29:53,120 --> 00:29:48,150 possibility is something that I refer to 749 00:29:55,100 --> 00:29:53,130 as cosmic a bitumen you first study 750 00:29:56,600 --> 00:29:55,110 something in a novel way it's entirely 751 00:30:02,480 --> 00:29:56,610 novel it's never been observed before 752 00:30:04,310 --> 00:30:02,490 and so it's novelty is as we know you 753 00:30:05,390 --> 00:30:04,320 know conscious and one of the critical 754 00:30:06,830 --> 00:30:05,400 aspects of consciousness is very 755 00:30:09,140 --> 00:30:06,840 sensitive to novel things when I first 756 00:30:11,120 --> 00:30:09,150 put my finger on my arm I feel it after 757 00:30:12,800 --> 00:30:11,130 a while it obituaries in some manner 758 00:30:14,450 --> 00:30:12,810 there may be some kind of habituation is 759 00:30:15,980 --> 00:30:14,460 happening perhaps in some sort of 760 00:30:18,200 --> 00:30:15,990 collective consciousness when 761 00:30:21,590 --> 00:30:18,210 individuals making an observation at 762 00:30:23,630 --> 00:30:21,600 first it is novel to reality and then as 763 00:30:24,860 --> 00:30:23,640 it becomes repeated it becomes less and 764 00:30:27,560 --> 00:30:24,870 less novel as some sort of future 765 00:30:28,880 --> 00:30:27,570 situation in some sense this is sort of 766 00:30:30,770 --> 00:30:28,890 the opposite of morphic resonance 767 00:30:32,150 --> 00:30:30,780 morphic resonance is the idea that once 768 00:30:34,040 --> 00:30:32,160 you put something out there it expands 769 00:30:38,240 --> 00:30:34,050 here the notion is that when you put 770 00:30:39,800 --> 00:30:38,250 something out there it it declines when 771 00:30:42,080 --> 00:30:39,810 I first mentioned this to Rupert 772 00:30:44,210 --> 00:30:42,090 Sheldrake he was rather skeptical about 773 00:30:45,410 --> 00:30:44,220 it but after the New Yorker he contacted 774 00:30:47,690 --> 00:30:45,420 me and said that in fact he'd been 775 00:30:49,640 --> 00:30:47,700 experiencing decline effects a number of 776 00:30:51,350 --> 00:30:49,650 his paradigms as well so he's become a 777 00:30:52,490 --> 00:30:51,360 little bit more open to the possibility 778 00:30:54,080 --> 00:30:52,500 that there could be a flip side to 779 00:30:55,790 --> 00:30:54,090 morphic resonance which would be 780 00:30:59,330 --> 00:30:55,800 something along the lines of cosmic 781 00:31:00,680 --> 00:30:59,340 capitulum ffice ice that this these are 782 00:31:03,410 --> 00:31:00,690 not explanations these are just sort of 783 00:31:04,880 --> 00:31:03,420 like possible things that this decline 784 00:31:09,170 --> 00:31:04,890 effect reminds me of this almost 785 00:31:11,480 --> 00:31:09,180 metaphorical is a imagine that you have 786 00:31:14,060 --> 00:31:11,490 a telescope and you pointed at a distant 787 00:31:15,470 --> 00:31:14,070 object and it's not in focus initially 788 00:31:17,450 --> 00:31:15,480 that distant object is going to be 789 00:31:19,610 --> 00:31:17,460 blurry and when it's blurry it's going 790 00:31:22,040 --> 00:31:19,620 to include more visual angle it's going 791 00:31:22,580 --> 00:31:22,050 to be larger as you bring it into focus 792 00:31:24,830 --> 00:31:22,590 and relax 793 00:31:27,049 --> 00:31:24,840 get smaller right it'll be sharper but 794 00:31:28,820 --> 00:31:27,059 smaller it may be that when we do 795 00:31:32,390 --> 00:31:28,830 scientific research it's kind of the 796 00:31:34,130 --> 00:31:32,400 equivalent of printing the metaphorical 797 00:31:36,169 --> 00:31:34,140 telescope at the phenomena when you 798 00:31:38,870 --> 00:31:36,179 first look at it the phenomena is sort 799 00:31:40,190 --> 00:31:38,880 of more fuzzy and distributed but then 800 00:31:42,950 --> 00:31:40,200 as you do more research 801 00:31:44,720 --> 00:31:42,960 it becomes more narrow which means that 802 00:31:46,850 --> 00:31:44,730 areas that used to be occluded or that 803 00:31:48,320 --> 00:31:46,860 the object is no longer there and so it 804 00:31:50,299 --> 00:31:48,330 may be that there is some sort of way in 805 00:31:52,940 --> 00:31:50,309 which the act of observation by becoming 806 00:31:55,370 --> 00:31:52,950 more precise actually areas that where 807 00:31:57,049 --> 00:31:55,380 the effect was once it's no longer there 808 00:31:58,610 --> 00:31:57,059 this also suggests that if you really 809 00:32:00,740 --> 00:31:58,620 get the bull's eye 810 00:32:02,480 --> 00:32:00,750 that it'll be there even in spades so 811 00:32:03,950 --> 00:32:02,490 there it sort of gives hope that it that 812 00:32:06,019 --> 00:32:03,960 it's just a matter of getting it exactly 813 00:32:07,490 --> 00:32:06,029 right but the idea is that typically 814 00:32:09,649 --> 00:32:07,500 when we're you know we're sort of 815 00:32:11,899 --> 00:32:09,659 throwing darts at things we're just 816 00:32:14,240 --> 00:32:11,909 getting right on the periphery and not 817 00:32:19,580 --> 00:32:14,250 really narrowing it down to its absolute 818 00:32:21,200 --> 00:32:19,590 fundamental locus and then the third 819 00:32:23,480 --> 00:32:21,210 thing and this is again not even really 820 00:32:25,460 --> 00:32:23,490 an explanation but it's just sort of an 821 00:32:27,620 --> 00:32:25,470 intuition about what's going on here is 822 00:32:29,330 --> 00:32:27,630 that beginner's luck there may be 823 00:32:31,580 --> 00:32:29,340 something about this notion of 824 00:32:35,480 --> 00:32:31,590 beginner's luck that somehow intrinsic 825 00:32:38,450 --> 00:32:35,490 in the way that reality is is is 826 00:32:41,210 --> 00:32:38,460 structured that when we do things the 827 00:32:43,370 --> 00:32:41,220 first time they are they afford a 828 00:32:46,220 --> 00:32:43,380 greater opportunity than later on who 829 00:32:48,200 --> 00:32:46,230 knows why that would be but it just 830 00:32:49,730 --> 00:32:48,210 seems possible that beginner's luck is 831 00:32:52,700 --> 00:32:49,740 an intrinsic aspect of reality that 832 00:32:55,279 --> 00:32:52,710 certainly is the claim that Kolo made in 833 00:32:58,940 --> 00:32:55,289 the opening quote so in terms of future 834 00:33:01,220 --> 00:32:58,950 directions first it suggests that novel 835 00:33:03,230 --> 00:33:01,230 paradigms should do better than ones 836 00:33:04,970 --> 00:33:03,240 that are strict replications and 837 00:33:06,380 --> 00:33:04,980 secondly I don't really have time to go 838 00:33:07,850 --> 00:33:06,390 into this one of the things that I've 839 00:33:10,519 --> 00:33:07,860 really been pushing hard for us we need 840 00:33:12,380 --> 00:33:10,529 to have an open source data repository 841 00:33:13,940 --> 00:33:12,390 where people log their experiments ahead 842 00:33:17,360 --> 00:33:13,950 of time and report the outcome 843 00:33:20,480 --> 00:33:17,370 regardless of results afterwards the 844 00:33:23,149 --> 00:33:20,490 problem in the current system is that we 845 00:33:25,220 --> 00:33:23,159 don't know how to fit the positive 846 00:33:26,419 --> 00:33:25,230 results with respect to all the larger 847 00:33:28,720 --> 00:33:26,429 negative results that are out there and 848 00:33:31,370 --> 00:33:28,730 are not being reported we need to 849 00:33:33,889 --> 00:33:31,380 scientists need to be required to make 850 00:33:35,779 --> 00:33:33,899 all of their research available and not 851 00:33:38,690 --> 00:33:35,789 just that which they've able in 852 00:33:40,909 --> 00:33:38,700 able to spin for acceptance into the 853 00:33:43,279 --> 00:33:40,919 scientific record it's only by knowing 854 00:33:45,950 --> 00:33:43,289 the full set of data that we'll be able 855 00:33:47,389 --> 00:33:45,960 to understand how the positive results 856 00:33:50,239 --> 00:33:47,399 that we get relate to the negative 857 00:33:52,690 --> 00:33:50,249 results that may also be out there so my 858 00:33:55,129 --> 00:33:52,700 final thoughts are two the first is it's 859 00:33:56,419 --> 00:33:55,139 ironically it may be that the domain of 860 00:33:58,249 --> 00:33:56,429 investigation that has been most 861 00:34:00,590 --> 00:33:58,259 frequently accused of a lack of rigor 862 00:34:02,299 --> 00:34:00,600 SCI research will be the impetus for 863 00:34:04,549 --> 00:34:02,309 reining in the sloppy practices of 864 00:34:08,770 --> 00:34:04,559 science as a whole and I think this is I 865 00:34:11,930 --> 00:34:08,780 think this is really a possibility and 866 00:34:14,960 --> 00:34:11,940 secondly I suspect that the best 867 00:34:17,299 --> 00:34:14,970 evidence for SCI may not be found at the 868 00:34:19,250 --> 00:34:17,309 individuals experimental level but 869 00:34:21,530 --> 00:34:19,260 rather at the meta level that when we 870 00:34:23,210 --> 00:34:21,540 set up this giant repository we will 871 00:34:25,129 --> 00:34:23,220 then be able to actually see these 872 00:34:27,109 --> 00:34:25,139 anomalies and they won't be able to be 873 00:34:29,000 --> 00:34:27,119 dismissed so it really may be the meta 874 00:34:30,740 --> 00:34:29,010 level where we're really gonna find the 875 00:34:41,359 --> 00:34:30,750 best evidence for SCI thank you very 876 00:34:46,760 --> 00:34:41,369 much I think you're gonna get a long 877 00:34:49,039 --> 00:34:46,770 line here I've done healing research for 878 00:34:51,109 --> 00:34:49,049 a number of decades now and I offer 879 00:34:53,329 --> 00:34:51,119 badly from a lack of a decline effect 880 00:34:54,889 --> 00:34:53,339 and this has bugged me I mean quite 881 00:34:56,569 --> 00:34:54,899 frankly it sounds silly but it's 882 00:34:59,960 --> 00:34:56,579 actually bugged me so at this point I've 883 00:35:02,000 --> 00:34:59,970 done 12 mice experiments on cancer that 884 00:35:04,609 --> 00:35:02,010 has 100% fatality and I don't have any 885 00:35:05,660 --> 00:35:04,619 decline effect at all in the beginning 886 00:35:07,910 --> 00:35:05,670 of your talk you were talking about 887 00:35:09,260 --> 00:35:07,920 changing things like colour and this and 888 00:35:10,609 --> 00:35:09,270 and and so you change something and then 889 00:35:11,030 --> 00:35:10,619 it goes down you change something goes 890 00:35:12,620 --> 00:35:11,040 down 891 00:35:14,750 --> 00:35:12,630 the flipside that you're making me 892 00:35:16,490 --> 00:35:14,760 consider is that the most number of 893 00:35:18,650 --> 00:35:16,500 studies I've ever done in one place is 894 00:35:20,270 --> 00:35:18,660 three I've actually done it in six 895 00:35:23,000 --> 00:35:20,280 places I'm wondering if changing the 896 00:35:27,220 --> 00:35:23,010 actual place might mitigate somehow the 897 00:35:30,349 --> 00:35:27,230 decline effect that's interesting I've 898 00:35:31,700 --> 00:35:30,359 I've found that I can't replicate 899 00:35:33,160 --> 00:35:31,710 certain effects but then when other 900 00:35:36,049 --> 00:35:33,170 people try them for the first time 901 00:35:37,880 --> 00:35:36,059 sometimes they can get it and then they 902 00:35:41,900 --> 00:35:37,890 suffer that another they their own local 903 00:35:43,490 --> 00:35:41,910 decline effect so it there may be when 904 00:35:45,950 --> 00:35:43,500 all the dust settles some sort of 905 00:35:47,870 --> 00:35:45,960 peculiar rules about what the sort of 906 00:35:49,200 --> 00:35:47,880 geography that you know you get a local 907 00:35:51,780 --> 00:35:49,210 decline effect and then a glow 908 00:35:54,180 --> 00:35:51,790 decline effect and that's certainly 909 00:35:56,550 --> 00:35:54,190 possible we just need to do a lot more 910 00:35:59,340 --> 00:35:56,560 research in the parameters to be able to 911 00:36:01,500 --> 00:35:59,350 establish those kind of rules I think 912 00:36:03,570 --> 00:36:01,510 the problem is that in science the ideal 913 00:36:05,370 --> 00:36:03,580 is that you replicate an experiment 914 00:36:07,320 --> 00:36:05,380 perfectly but I have a little statement 915 00:36:09,600 --> 00:36:07,330 I wrote so you have to read here no 916 00:36:12,060 --> 00:36:09,610 experiment is perfectly isolated and 917 00:36:14,310 --> 00:36:12,070 perfectly replicable because the 918 00:36:16,710 --> 00:36:14,320 universe changes around the experiment 919 00:36:17,220 --> 00:36:16,720 no matter how hard we try to object 920 00:36:19,590 --> 00:36:17,230 avait 921 00:36:21,690 --> 00:36:19,600 isolate the experiment from the universe 922 00:36:23,760 --> 00:36:21,700 now we've all been saying for the past 923 00:36:26,670 --> 00:36:23,770 few days that all things are connected 924 00:36:28,500 --> 00:36:26,680 when it comes to size of X then this 925 00:36:31,470 --> 00:36:28,510 change in the universe should be taken 926 00:36:33,480 --> 00:36:31,480 into account regarding testing any side 927 00:36:35,610 --> 00:36:33,490 effects in other words why I'm 928 00:36:38,700 --> 00:36:35,620 suggesting is that there's a certain 929 00:36:42,650 --> 00:36:38,710 half-life just due to the normal change 930 00:36:45,390 --> 00:36:42,660 in the universe that we see as time and 931 00:36:50,190 --> 00:36:45,400 this should be taken account into all 932 00:36:52,230 --> 00:36:50,200 sigh experiments it is the idea that the 933 00:36:54,180 --> 00:36:52,240 universe is changing in a sense what 934 00:36:56,430 --> 00:36:54,190 we're trying to do with SCI experiments 935 00:36:59,220 --> 00:36:56,440 we're trying to bend the universe around 936 00:37:00,750 --> 00:36:59,230 any individual experiment but the 937 00:37:02,490 --> 00:37:00,760 universe is changing around the 938 00:37:04,740 --> 00:37:02,500 experiment at the same time and we have 939 00:37:08,040 --> 00:37:04,750 to take this into account and perhaps 940 00:37:10,410 --> 00:37:08,050 people who have psychics and such have 941 00:37:12,530 --> 00:37:10,420 success after success if it's possible 942 00:37:15,120 --> 00:37:12,540 with side-effects or somehow 943 00:37:16,970 --> 00:37:15,130 subconsciously taking this change in the 944 00:37:19,740 --> 00:37:16,980 universe into effect from time to time 945 00:37:22,350 --> 00:37:19,750 well I think that's certainly something 946 00:37:24,480 --> 00:37:22,360 to consider if you imagine that the 947 00:37:29,760 --> 00:37:24,490 universe is sort of gradually changing 948 00:37:33,630 --> 00:37:29,770 and that changes in in changes in the 949 00:37:35,040 --> 00:37:33,640 universe are associated with reduction 950 00:37:37,320 --> 00:37:35,050 in the client effects then that could 951 00:37:39,830 --> 00:37:37,330 also explain why you see these recovery 952 00:37:42,090 --> 00:37:39,840 effects that it may be that over time 953 00:37:43,230 --> 00:37:42,100 there the experiment is even though 954 00:37:44,820 --> 00:37:43,240 you're seemingly doing the same 955 00:37:49,580 --> 00:37:44,830 experiment you're not really and that's 956 00:37:53,610 --> 00:37:52,230 that that certainly certainly could be 957 00:37:55,980 --> 00:37:53,620 the case I wish the universe would 958 00:37:57,810 --> 00:37:55,990 change faster so that my decline effects 959 00:38:00,480 --> 00:37:57,820 didn't happen so quickly would be slower 960 00:38:02,240 --> 00:38:00,490 right kicking ass a stir I wish we could 961 00:38:08,390 --> 00:38:02,250 ask the questions faster so we 962 00:38:10,790 --> 00:38:08,400 get more there there are three factors 963 00:38:12,860 --> 00:38:10,800 that could be playing on the size effect 964 00:38:15,230 --> 00:38:12,870 that comes first quite high and then 965 00:38:17,480 --> 00:38:15,240 comes down one could be the real size 966 00:38:19,970 --> 00:38:17,490 effect you might say and the other could 967 00:38:22,220 --> 00:38:19,980 be two possible augmentations of that 968 00:38:24,200 --> 00:38:22,230 size effect and two of the candidates 969 00:38:27,920 --> 00:38:24,210 for those augmentations are the 970 00:38:30,350 --> 00:38:27,930 experimenter effect and the universe the 971 00:38:32,300 --> 00:38:30,360 it could be that the experimenter is not 972 00:38:34,850 --> 00:38:32,310 the same experimenter and that whatever 973 00:38:37,370 --> 00:38:34,860 constitutes the experimenter effect has 974 00:38:39,850 --> 00:38:37,380 it itself has declined just the 975 00:38:42,620 --> 00:38:39,860 enthusiasm or the surprise on the 976 00:38:45,230 --> 00:38:42,630 staking your whole career on it but the 977 00:38:47,120 --> 00:38:45,240 universe and a lot of the papers that 978 00:38:48,800 --> 00:38:47,130 we've been hearing imply that there's a 979 00:38:51,680 --> 00:38:48,810 kind of a cooperation between the 980 00:38:54,800 --> 00:38:51,690 universe and what we perceive or can do 981 00:38:57,260 --> 00:38:54,810 and there's no reason why whatever the 982 00:39:00,170 --> 00:38:57,270 universe refers to why it should give a 983 00:39:03,440 --> 00:39:00,180 damn after a while to be at your service 984 00:39:05,630 --> 00:39:03,450 as an experimenter to turn you in the 985 00:39:07,370 --> 00:39:05,640 right direction if you've turned in the 986 00:39:10,010 --> 00:39:07,380 right direction fine if you haven't fine 987 00:39:15,770 --> 00:39:10,020 it's just gonna poop out and go augment 988 00:39:17,810 --> 00:39:15,780 somebody else's research yeah I you know 989 00:39:21,260 --> 00:39:17,820 I I can't speak for the I cannot speak 990 00:39:24,940 --> 00:39:23,150 but but what I can't say is is that you 991 00:39:26,630 --> 00:39:24,950 mentioned a really important 992 00:39:27,920 --> 00:39:26,640 interpretation of the client fact that I 993 00:39:31,100 --> 00:39:27,930 didn't mention and this is one that I 994 00:39:33,020 --> 00:39:31,110 know Dean favors and Daryl BEM which is 995 00:39:36,080 --> 00:39:33,030 the idea that it's driven by the 996 00:39:37,580 --> 00:39:36,090 enthusiasm of the experimenter than when 997 00:39:39,320 --> 00:39:37,590 experimenters first start running their 998 00:39:42,080 --> 00:39:39,330 studies that they're really gung-ho and 999 00:39:44,690 --> 00:39:42,090 and somehow that gets communicated to 1000 00:39:45,740 --> 00:39:44,700 the participant I just have to emphasize 1001 00:39:47,660 --> 00:39:45,750 in there I mean some of these studies 1002 00:39:49,430 --> 00:39:47,670 we've run them there's very little 1003 00:39:51,050 --> 00:39:49,440 interaction with the experiment or 1004 00:39:52,430 --> 00:39:51,060 they're basically interacting just for 1005 00:39:53,890 --> 00:39:52,440 the computer and we still see the client 1006 00:39:56,300 --> 00:39:53,900 effect so well I think that the 1007 00:39:59,750 --> 00:39:56,310 enthusiasm of the experimenter may be 1008 00:40:00,980 --> 00:39:59,760 part of it I I'm not persuaded that that 1009 00:40:04,300 --> 00:40:00,990 alone would explain it 1010 00:40:06,920 --> 00:40:04,310 you've largely just answered my question 1011 00:40:10,040 --> 00:40:06,930 we've seen all over the place decline 1012 00:40:12,980 --> 00:40:10,050 effects where the interaction between 1013 00:40:14,510 --> 00:40:12,990 the experimenter and the experiment even 1014 00:40:15,740 --> 00:40:14,520 if it's just a computer running I don't 1015 00:40:17,780 --> 00:40:15,750 know if you saw my 1016 00:40:20,810 --> 00:40:17,790 last night where we see this you know 1017 00:40:23,560 --> 00:40:20,820 huge decline effect in a compute pure 1018 00:40:26,270 --> 00:40:23,570 computer experiment but we really can 1019 00:40:27,950 --> 00:40:26,280 connect it with our own enthusiasm and 1020 00:40:30,980 --> 00:40:27,960 that does seem to be the most 1021 00:40:32,960 --> 00:40:30,990 parsimonious answer to the decline 1022 00:40:36,109 --> 00:40:32,970 effect it does mean that all of these 1023 00:40:37,790 --> 00:40:36,119 experiments involve sigh but that's not 1024 00:40:40,190 --> 00:40:37,800 a far reach from what we've been seeing 1025 00:40:45,520 --> 00:40:40,200 so I'm surprised that that's not your 1026 00:40:50,030 --> 00:40:45,530 primary thought well I mean I would say 1027 00:40:52,070 --> 00:40:50,040 it's it's hard for me to believe let me 1028 00:40:53,599 --> 00:40:52,080 just be emphasized I am open to all 1029 00:40:54,770 --> 00:40:53,609 possible explanations there are two 1030 00:40:58,550 --> 00:40:54,780 different versions of the enthusiasm 1031 00:41:00,020 --> 00:40:58,560 account one is a non-conventional 1032 00:41:02,510 --> 00:41:00,030 account which is just that the 1033 00:41:04,810 --> 00:41:02,520 participant themselves is picking up 1034 00:41:07,130 --> 00:41:04,820 directly on an interaction with the 1035 00:41:08,960 --> 00:41:07,140 researcher and that's having its effect 1036 00:41:12,710 --> 00:41:08,970 the other possibility is that somehow 1037 00:41:15,410 --> 00:41:12,720 the researchers enthusiasm is making its 1038 00:41:19,640 --> 00:41:15,420 way into the ether and that's driving 1039 00:41:22,970 --> 00:41:19,650 the effect that the second possibility 1040 00:41:26,630 --> 00:41:22,980 is certainly there it's it's not obvious 1041 00:41:28,820 --> 00:41:26,640 to me that researchers enthusiasm for 1042 00:41:31,280 --> 00:41:28,830 effects should necessarily show a 1043 00:41:34,250 --> 00:41:31,290 decline I would I mean I've seen my 1044 00:41:36,440 --> 00:41:34,260 postdoc Michael Franklin you know the 1045 00:41:38,120 --> 00:41:36,450 longer that he's been involved in this 1046 00:41:40,010 --> 00:41:38,130 he's invested more so you could easily 1047 00:41:41,359 --> 00:41:40,020 make the case that the longer you've 1048 00:41:42,890 --> 00:41:41,369 been involved in research and you've 1049 00:41:44,450 --> 00:41:42,900 started presenting and stuff you should 1050 00:41:46,730 --> 00:41:44,460 be more excited about it because you've 1051 00:41:48,530 --> 00:41:46,740 really put your name on it so it's a 1052 00:41:50,630 --> 00:41:48,540 peculiar kind of enthusiasm that 1053 00:41:55,339 --> 00:41:50,640 dwindles as you've staked more of your 1054 00:41:57,470 --> 00:41:55,349 reputation on it with that view in mind 1055 00:41:59,930 --> 00:41:57,480 I think the word is intent it's not 1056 00:42:03,890 --> 00:41:59,940 necessarily enthusiasm it's focused 1057 00:42:06,380 --> 00:42:03,900 intent and how possibly how rooted we 1058 00:42:08,210 --> 00:42:06,390 are in a sense that there is only this 1059 00:42:10,160 --> 00:42:08,220 reality that we're experimenting with 1060 00:42:11,329 --> 00:42:10,170 there have been a whole bunch of 1061 00:42:14,420 --> 00:42:11,339 researchers and a whole bunch of guys 1062 00:42:16,940 --> 00:42:14,430 one that comes to mind is a very strong 1063 00:42:18,530 --> 00:42:16,950 peak a guy named Michael Hutchison had 1064 00:42:20,390 --> 00:42:18,540 an effect named after him Hutchison in 1065 00:42:24,380 --> 00:42:20,400 fact he could do certain things with his 1066 00:42:26,359 --> 00:42:24,390 machines that nobody else could and this 1067 00:42:28,099 --> 00:42:26,369 is just if you think about it what is 1068 00:42:28,819 --> 00:42:28,109 the nature of the Year Ally that your 1069 00:42:30,859 --> 00:42:28,829 work 1070 00:42:32,660 --> 00:42:30,869 and then you have some researchers like 1071 00:42:34,540 --> 00:42:32,670 William tiller and there's earlier 1072 00:42:37,339 --> 00:42:34,550 speaker URI con talked about 1073 00:42:39,109 --> 00:42:37,349 conditioning space and altering the 1074 00:42:42,020 --> 00:42:39,119 space that you're working in so that it 1075 00:42:44,780 --> 00:42:42,030 no longer behaves the way quote normal 1076 00:42:47,030 --> 00:42:44,790 whatever that is space space and so I'd 1077 00:42:50,569 --> 00:42:47,040 say the focus of intent by the 1078 00:42:53,000 --> 00:42:50,579 researcher can flag the enthusiasm may 1079 00:42:54,290 --> 00:42:53,010 be there but there is a real and there 1080 00:42:56,780 --> 00:42:54,300 are tons of shamanic and other 1081 00:42:58,609 --> 00:42:56,790 traditions were where this is a key to 1082 00:43:01,370 --> 00:42:58,619 manifesting the reality that you really 1083 00:43:03,920 --> 00:43:01,380 want so I know that's fairly far out but 1084 00:43:07,220 --> 00:43:03,930 well we're in this is the right group 1085 00:43:09,290 --> 00:43:07,230 for that I just want to say that this 1086 00:43:11,329 --> 00:43:09,300 notion is sort of in that's getting 1087 00:43:13,910 --> 00:43:11,339 close to the notion of habituation which 1088 00:43:15,470 --> 00:43:13,920 is the sort of variation on the cosmic 1089 00:43:18,020 --> 00:43:15,480 thing but I think you're right it could 1090 00:43:22,609 --> 00:43:18,030 well be driven just by the individual as 1091 00:43:26,059 --> 00:43:22,619 opposed to buy some more global kind of 1092 00:43:27,980 --> 00:43:26,069 a situation so since there are since 1093 00:43:30,530 --> 00:43:27,990 arbitrary random processes can't have 1094 00:43:33,530 --> 00:43:30,540 arbitrarily long what you might consider 1095 00:43:36,140 --> 00:43:33,540 hits right is there a anti symmetric 1096 00:43:37,849 --> 00:43:36,150 effect here so in the sense that if you 1097 00:43:39,380 --> 00:43:37,859 had an experiment that was negative at 1098 00:43:42,589 --> 00:43:39,390 first it's more likely that you're not 1099 00:43:44,990 --> 00:43:42,599 going to consider it again so that file 1100 00:43:46,579 --> 00:43:45,000 drawer effect seems to me maybe make 1101 00:43:49,190 --> 00:43:46,589 sense to me that there is no decline 1102 00:43:52,700 --> 00:43:49,200 effect and that the total sample space 1103 00:43:55,160 --> 00:43:52,710 of experiments isn't robust absolutely 1104 00:43:57,349 --> 00:43:55,170 that that is one of the major 1105 00:43:59,870 --> 00:43:57,359 explanations to be and that's why we 1106 00:44:02,359 --> 00:43:59,880 need to have this open database 1107 00:44:04,520 --> 00:44:02,369 repository because according to that 1108 00:44:08,210 --> 00:44:04,530 interpretation there should be a huge 1109 00:44:11,059 --> 00:44:08,220 number of failed first experiments that 1110 00:44:12,620 --> 00:44:11,069 you know are not or not pursued and so 1111 00:44:14,569 --> 00:44:12,630 what you really need to do is to look at 1112 00:44:16,730 --> 00:44:14,579 the frequency with which you know 1113 00:44:18,740 --> 00:44:16,740 initial experiments are failing versus 1114 00:44:20,329 --> 00:44:18,750 succeeding and you could do Monte Carlo 1115 00:44:22,809 --> 00:44:20,339 simulations to see whether or not that 1116 00:44:24,710 --> 00:44:22,819 would be sufficient to produce the 1117 00:44:26,660 --> 00:44:24,720 magnitude of the client effects that we 1118 00:44:29,059 --> 00:44:26,670 get but the only way we can do that is 1119 00:44:31,069 --> 00:44:29,069 by knowing situating the positive 1120 00:44:32,480 --> 00:44:31,079 results within the context of all the 1121 00:44:34,790 --> 00:44:32,490 negative results that may or may not be 1122 00:44:36,800 --> 00:44:34,800 out there thank you he pretty much stole 1123 00:44:39,380 --> 00:44:36,810 my thunder my my question was gonna be 1124 00:44:40,700 --> 00:44:39,390 you can only have a decline effect if 1125 00:44:42,170 --> 00:44:40,710 you already had a positive correlation 1126 00:44:43,760 --> 00:44:42,180 to begin with 1127 00:44:45,170 --> 00:44:43,770 has anyone conducted experiments where 1128 00:44:47,510 --> 00:44:45,180 they did not have a positive or negative 1129 00:44:49,640 --> 00:44:47,520 correlation continue that experiment to 1130 00:44:52,120 --> 00:44:49,650 see if you had and absolutely and then 1131 00:44:54,170 --> 00:44:52,130 it declining right this is another 1132 00:44:55,730 --> 00:44:54,180 approach that I think would be very 1133 00:44:57,650 --> 00:44:55,740 useful what I think would be a great 1134 00:45:01,250 --> 00:44:57,660 idea so let me put this out here right 1135 00:45:02,600 --> 00:45:01,260 now is to come up with 25 experiments 1136 00:45:04,760 --> 00:45:02,610 that have not been run that are novel 1137 00:45:06,980 --> 00:45:04,770 that ought to maybe work given what we 1138 00:45:09,710 --> 00:45:06,990 know but have not been done and then get 1139 00:45:11,300 --> 00:45:09,720 the group of researchers to commit to 1140 00:45:14,690 --> 00:45:11,310 running each one of those experiments 1141 00:45:17,120 --> 00:45:14,700 ten times regardless of whether or not 1142 00:45:20,540 --> 00:45:17,130 they worked the first time 1143 00:45:22,910 --> 00:45:20,550 and then look to see the pattern of of 1144 00:45:24,350 --> 00:45:22,920 significance the decline effect account 1145 00:45:26,240 --> 00:45:24,360 would predict that you should be more 1146 00:45:28,070 --> 00:45:26,250 likely to get positive results in the 1147 00:45:30,080 --> 00:45:28,080 first trial than the fifth trial but 1148 00:45:31,910 --> 00:45:30,090 that is exactly the kind of paradigm 1149 00:45:33,980 --> 00:45:31,920 that's required and it requires people 1150 00:45:35,570 --> 00:45:33,990 to carry on doing the studies repeatedly 1151 00:45:37,750 --> 00:45:35,580 even when they don't get the effects 1152 00:45:42,560 --> 00:45:37,760 initially exactly exactly 1153 00:45:45,290 --> 00:45:42,570 thrown out the window exactly you 1154 00:45:48,260 --> 00:45:45,300 presented it somewhat facetiously and 1155 00:45:50,540 --> 00:45:48,270 the audience laughed but I am actually 1156 00:45:52,910 --> 00:45:50,550 rather charmed by your proposal that the 1157 00:45:55,370 --> 00:45:52,920 decline effect could actually decline do 1158 00:45:57,770 --> 00:45:55,380 you have any estimate for how big a 1159 00:46:00,110 --> 00:45:57,780 research effort on the decline effect 1160 00:46:01,850 --> 00:46:00,120 itself would be needed in order to run 1161 00:46:03,610 --> 00:46:01,860 it into the ground so that we don't need 1162 00:46:08,240 --> 00:46:03,620 to worry about it any more 1163 00:46:10,910 --> 00:46:08,250 I was only half facetiously I I also 1164 00:46:13,100 --> 00:46:10,920 think that the best hope of getting rid 1165 00:46:16,880 --> 00:46:13,110 of the decline effect is by studying and 1166 00:46:19,190 --> 00:46:16,890 I have no idea how much effort that 1167 00:46:22,760 --> 00:46:19,200 would require but that's certainly where 1168 00:46:24,140 --> 00:46:22,770 some of my energy is going I am I was 1169 00:46:26,270 --> 00:46:24,150 wondering if you could talk a little bit 1170 00:46:27,920 --> 00:46:26,280 more about sort of the local versus 1171 00:46:30,500 --> 00:46:27,930 global decline effects and what you know 1172 00:46:33,020 --> 00:46:30,510 about that and then secondly kind of in 1173 00:46:35,060 --> 00:46:33,030 conjunction if something about having 1174 00:46:36,920 --> 00:46:35,070 information about one's own experiments 1175 00:46:39,380 --> 00:46:36,930 drives the decline effect in some way 1176 00:46:42,770 --> 00:46:39,390 could that not be a problem for this 1177 00:46:44,570 --> 00:46:42,780 sort of term metadata idea where when we 1178 00:46:46,370 --> 00:46:44,580 now have all the information available 1179 00:46:49,070 --> 00:46:46,380 about everybody's possible experiments 1180 00:46:50,900 --> 00:46:49,080 it's somehow it's harder to have your 1181 00:46:54,060 --> 00:46:50,910 own local version of it decline effect 1182 00:46:58,620 --> 00:46:56,700 most of that is just needs to be studied 1183 00:47:00,840 --> 00:46:58,630 I mean I think we really need to go I 1184 00:47:02,790 --> 00:47:00,850 think we need to admit that the client 1185 00:47:03,780 --> 00:47:02,800 effect is an area that deserve studying 1186 00:47:08,070 --> 00:47:03,790 and then look at it very systematically 1187 00:47:11,220 --> 00:47:08,080 both at the local and at the and at the 1188 00:47:12,930 --> 00:47:11,230 global level I also think it's a very 1189 00:47:14,310 --> 00:47:12,940 intriguing possibility what triggers the 1190 00:47:17,310 --> 00:47:14,320 decline effect is that the running of 1191 00:47:20,910 --> 00:47:17,320 the experiment or is it the analyzing of 1192 00:47:21,930 --> 00:47:20,920 the data and what I think is quite if if 1193 00:47:23,910 --> 00:47:21,940 we're gonna go with some of the 1194 00:47:25,830 --> 00:47:23,920 non-conventional accounts you know the 1195 00:47:28,170 --> 00:47:25,840 sort of Heisenberg thing what may really 1196 00:47:29,370 --> 00:47:28,180 Drive the effect is not the point in 1197 00:47:31,850 --> 00:47:29,380 which it's collected but the point in 1198 00:47:34,350 --> 00:47:31,860 which the experimenter looks at the 1199 00:47:37,200 --> 00:47:34,360 results and that's the kind of thing you 1200 00:47:39,300 --> 00:47:37,210 could also systematically vary you know 1201 00:47:42,210 --> 00:47:39,310 when do they peak and there's this I 1202 00:47:43,950 --> 00:47:42,220 there's the possibility that you want to 1203 00:47:45,720 --> 00:47:43,960 be really it may be that the peaking is 1204 00:47:47,400 --> 00:47:45,730 really the problem here so you know 1205 00:47:49,350 --> 00:47:47,410 really collect a lot of end before you 1206 00:47:51,480 --> 00:47:49,360 peak because once you peak that's when 1207 00:47:53,160 --> 00:47:51,490 the effect starts going down those kinds 1208 00:47:55,410 --> 00:47:53,170 of things systematically controlling 1209 00:47:57,480 --> 00:47:55,420 when the peak is done and who's speaking 1210 00:47:59,700 --> 00:47:57,490 how many people are peaking might be one 1211 00:48:00,740 --> 00:47:59,710 way of finding some of the side effects 1212 00:48:03,570 --> 00:48:00,750 at the meta level 1213 00:48:06,350 --> 00:48:03,580 well somebody who's quite familiar 1214 00:48:09,150 --> 00:48:06,360 painfully familiar with decline effects 1215 00:48:12,870 --> 00:48:09,160 we studied it for many years over many 1216 00:48:17,940 --> 00:48:12,880 different experiments it's quite robust 1217 00:48:19,890 --> 00:48:17,950 oh here it does seem to occur at local 1218 00:48:21,920 --> 00:48:19,900 as well as middle levels and 1219 00:48:24,510 --> 00:48:21,930 incidentally we also looked at 1220 00:48:29,010 --> 00:48:24,520 experiments that had no effect and there 1221 00:48:32,130 --> 00:48:29,020 was no decline they were pretty a pretty 1222 00:48:36,240 --> 00:48:32,140 level one thing that I think is worth 1223 00:48:38,700 --> 00:48:36,250 mentioning is the the oscillations one 1224 00:48:41,550 --> 00:48:38,710 season the affected it's very seldom a 1225 00:48:46,280 --> 00:48:41,560 direct one-to-one decline what you're 1226 00:48:48,930 --> 00:48:46,290 seeing is a trend superimposed on 1227 00:48:53,010 --> 00:48:48,940 fluctuations that you would expect in a 1228 00:48:56,180 --> 00:48:53,020 random process and they do seem to 1229 00:48:59,610 --> 00:48:56,190 recover after a certain period of time 1230 00:49:01,640 --> 00:48:59,620 but this has led us to suspect going 1231 00:49:05,010 --> 00:49:01,650 back to our friend Verner Heisenberg 1232 00:49:07,310 --> 00:49:05,020 that maybe there's something in here 1233 00:49:10,100 --> 00:49:07,320 that relates to the concept of 1234 00:49:12,860 --> 00:49:10,110 uncertainty in fact we have a paper that 1235 00:49:17,840 --> 00:49:12,870 we called information and uncertainty 1236 00:49:21,070 --> 00:49:17,850 and I it may be that since we are 1237 00:49:24,170 --> 00:49:21,080 dealing with fundamentally probabilistic 1238 00:49:25,910 --> 00:49:24,180 processes which of course we are 1239 00:49:29,770 --> 00:49:25,920 whenever we're doing statistical 1240 00:49:34,340 --> 00:49:29,780 analysis that the more we do the more 1241 00:49:37,550 --> 00:49:34,350 data we collect the the the more the 1242 00:49:40,120 --> 00:49:37,560 uncertainty is reduced and the 1243 00:49:43,130 --> 00:49:40,130 uncertainty may be an inherently 1244 00:49:46,610 --> 00:49:43,140 important component of driving these 1245 00:49:49,070 --> 00:49:46,620 effects so as you reduce the uncertainty 1246 00:49:52,070 --> 00:49:49,080 and attempt to enhance the noise the 1247 00:49:54,530 --> 00:49:52,080 signal relative to the noise you may 1248 00:49:57,170 --> 00:49:54,540 actually be driving out the source of 1249 00:49:59,180 --> 00:49:57,180 the effect you're looking for and it may 1250 00:50:01,790 --> 00:49:59,190 just be that you get to the some point 1251 00:50:05,600 --> 00:50:01,800 in the decline where you say enough of 1252 00:50:08,150 --> 00:50:05,610 this maybe I was just messing with my 1253 00:50:10,250 --> 00:50:08,160 own head and there's nothing there and 1254 00:50:12,830 --> 00:50:10,260 you go back into a state of high 1255 00:50:16,630 --> 00:50:12,840 uncertainty hmm and my god there it is 1256 00:50:19,490 --> 00:50:16,640 again but I think this idea of 1257 00:50:22,820 --> 00:50:19,500 probability and uncertainty is worth 1258 00:50:28,520 --> 00:50:22,830 looking at because in informational 1259 00:50:31,760 --> 00:50:28,530 systems that have statistical basis this 1260 00:50:33,500 --> 00:50:31,770 could be a very fundamental issue and it 1261 00:50:36,620 --> 00:50:33,510 could help us understand the process 1262 00:50:41,270 --> 00:50:36,630 better yeah but I'm awfully glad to hear 1263 00:50:43,340 --> 00:50:41,280 we're not the only I bet that sounds 1264 00:50:46,520 --> 00:50:43,350 right to me the uncertainty feels like 1265 00:50:47,990 --> 00:50:46,530 it may play a role and certainly there's 1266 00:50:50,600 --> 00:50:48,000 a sense in which one gets at least a 1267 00:50:52,970 --> 00:50:50,610 false sense of certainty as one keeps 1268 00:50:54,260 --> 00:50:52,980 seeing positive results one thing for 1269 00:50:56,220 --> 00:50:54,270 certain we have to move on to the next 1270 00:50:59,220 --> 00:50:56,230 speaker so thank John you Jonathan